persistence follow-up gates: publication vs observed head, enforceable fences - #913

🏁 Script executed:

#!/bin/bash
# Description: Find every call site of FleetRecovery::checkpoint_bound.
rg -n -C3 '\bcheckpoint_bound\s*\(' --type=rust

Repository: AdaWorldAPI/lance-graph

Length of output: 161

🏁 Script executed:

#!/bin/bash
set -u

# Locate and inspect the relevant FleetRecovery code and nearby test module.
file="crates/lance-graph-supervisor/src/cycle_driver.rs"
if [ -f "$file" ]; then
  echo "== file size =="
  wc -l "$file"
  echo "== outline matches FleetRecovery/checkpoint =="
  ast-grep outline "$file" --match 'FleetRecovery|checkpoint_bound' --view expanded || true
  echo "== relevant source 760-920 =="
  sed -n '760,920p' "$file" | cat -n -v
  echo "== recovered_through occurrences =="
  rg -n -C3 '\brecovered_through\b|checkpoint_bound|foreign_min_cycle|after_cycle|scan_sealed' "$file"
else
  echo "missing $file"
fi

echo "== repository-wide FleetRecovery definitions/usages =="
rg -n -C2 '\bFleetRecovery\b|\bforeign_min_cycle\b|\brecovered_through\b|\back_cycle\b|checkpoint_bound' --type=rust || true

Repository: AdaWorldAPI/lance-graph

Length of output: 17674

🏁 Script executed:

#!/bin/bash
# Description: Inspect the Lance schema type used by guard_schema for a field-count accessor.
set -euo pipefail

rg -nP -C3 '\bfn\s+field\s*\(|\bpub\s+fields\b|\bfn\s+fields\s*\(' --type=rust -g '!target/**' | rg -i 'schema' | head -30

Repository: AdaWorldAPI/lance-graph

Length of output: 161

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== candidate files =="
fd -a 'cycle_sink\.rs|Cargo\.toml' . | sed 's#^\./##' | head -100

echo "== cycle_sink relevant sections =="
if fd -a 'cycle_sink\.rs' . | grep -q .; then
  f="$(fd 'cycle_sink\.rs' . | head -1)"
  echo "FILE=$f"
  wc -l "$f"
  sed -n '1,120p' "$f" | cat -n
  sed -n '280,380p' "$f" | cat -n
fi

echo "== lance deps =="
rg -n "lance|lance_dataset" Cargo.toml crates -g 'Cargo.toml' -C2 || true

echo "== schema usages =="
rg -n "guard_schema|cycle_store_schema|raw_append|ds\.append|append\(" crates/lance-graph/src -g '*.rs' -C2 || true

Repository: AdaWorldAPI/lance-graph

Length of output: 50380

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== cycle_store_schema and raw_append sections =="
cat -n crates/lance-graph/src/graph/cycle_sink.rs | sed -n '180,260p'
cat -n crates/lance-graph/src/graph/cycle_sink.rs 2>/tmp/sep.txt || true
cat -n crates/lance-graph/src/graph/cycle_sink.rs | sed -n '600,820p'
cat -n crates/lance-graph/src/graph/cycle_sink.rs | sed -n '820,960p'

echo "== Lockfile Lance Schema references =="
rg -n "name = \"lance\"|source =|lance-schema|Schema" Cargo.lock crates/lance-graph/Cargo.lock --max-count 80 || true

echo "== local vendored lance sources schema API =="
fd -a 'schema|Schema' Cargo.lock crates/lance-graph/Cargo.lock /tmp 2>/dev/null | head -1 || true
rg -n "struct Schema|impl Schema|pub fn fields|schema.fields" --glob '*.rs' --glob '!target/**' crates/lance-graph/src/graph/cycle_sink.rs crates/lance-graph/Cargo.lock 2>/dev/null || true

Repository: AdaWorldAPI/lance-graph

Length of output: 50380

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Cargo.lock Lance package entries (concise) =="
python3 - <<'PY'
from pathlib import Path
p=Path('Cargo.lock')
if not p.exists():
    print('Cargo.lock missing')
    raise SystemExit
text=p.read_text()
for i,blk in enumerate(text.split('\n[[package]]\n')):
    if i==0: blk='[package]'+'\n'+blk
    name='?'
    version='?'
    for line in blk.splitlines():
        if line.startswith('name = ') and 'lance' in line:
            name=line.split('=',1)[1].strip().strip('"')
        if line.startswith('version = '):
            version=line.split('=',1)[1].strip().strip('"')
    if 'lance' in name:
        print(f'{name} {version}')
PY

echo "== local registry/cargo cache packages if present =="
for d in "$CARGO_HOME/registry/src" "$HOME/.cargo/registry/src" "/usr/local/cargo/registry/src"; do
  [ -d "$d" ] || continue
  echo "$d"
  fd -a 'schema\.(rs|rs\.rs)$' "$d" -g '*lance*' | sed -n '1,20p' || true
  rg -n "crate_name = \"lance_schema\"|pub struct Schema|impl Schema|pub fn fields|fields\\(" "$d" -g '*lance*' -g '*.rs' | sed -n '1,80p' || true
done

echo "== Deterministic guard_schema behavior from source =="
python3 - <<'PY'
from pathlib import Path
src=Path('crates/lance-graph/src/graph/cycle_sink.rs').read_text()
start=src.index('/// Refuse a store whose schema')
end=src.index('\n    /// The highest durable cycle', start)
guard=src[start:end]
checks={
  'checks field names via got.field': 'got.field(field.name())' in guard,
  'checks data_type': 'g.data_type() != *field.data_type()' in guard,
  'checks nullable': 'g.nullable != field.is_nullable()' in guard,
  'recommends field count comparison': 'fields().len()' in guard or 'fields_len' in guard,
}
print('\n'.join(f'{k}: {v}' for k,v in checks.items()))
print(guard)
PY

Repository: AdaWorldAPI/lance-graph

Length of output: 1156

🌐 Web query:

Lance 9.0.0 Rust Dataset::open Schema fields accessor

💡 Result:

In Lance 9.0.0 (as part of the underlying Lance core and LanceDB ecosystem), you can access the dataset's schema by using the schema method on a Dataset object [1][2]. Once you have opened a dataset, the schema method returns a reference to a Schema object [1][2]. The Schema struct contains a fields accessor, which is a vector of Field objects (pub fields: Vec) [3]. Example usage: // Assuming you have an opened dataset variable let schema = dataset.schema; // Access the fields for field in &schema.fields { println!("Field name: {}, Data type: {:?}", field.name, field.data_type); } You can also use the field method on the Schema object to retrieve a specific field by its path, which handles nested field paths as well [3]. For operations involving LanceDB tables, note that tables have their own schema access patterns, typically via an async schema method on the Table object [4].

Citations:

• 1: https://docs.rs/lance/latest/lance/dataset/struct.Dataset.html
• 2: https://github.com/lancedb/lance/blob/119f87b3/rust/lance/src/dataset.rs
• 3: https://github.com/lancedb/lance/blob/119f87b3/rust/lance-core/src/datatypes/schema.rs
• 4: https://docs.rs/lancedb/latest/lancedb/table/struct.Table.html

🌐 Web query:

lance 9.0.0 Dataset append schema mismatch record batch schema

💡 Result:

In Lance (and LanceDB), a schema mismatch error during a dataset append operation typically occurs when the schema of the incoming record batch deviates from the existing dataset's registered schema [1][2][3]. The Lance write path performs strict compatibility checks when in append mode [1]. An error will be triggered if the incoming data: 1. Changes the data type of an existing column (e.g., attempting to append Float64 data to a column defined as Float32) [3]. 2. Alters the nullability or order of fields in a way that violates the established schema [1][2]. 3. Introduces structural differences, such as missing non-nullable columns or incompatible nested field definitions [1][2]. Specific technical scenarios leading to this error include: - Extension Type Handling: A known class of issues involves mismatch between Arrow extension types (like pa.json_()) and their underlying physical representations (e.g., LargeBinary) in the dataset [4][5]. Even when data content is logically correct, the metadata mismatch can cause the write path to reject the batch [4][5]. - Subschema Inserts: While recent versions have added support for inserting "subschemas" (allowing new rows to omit nullable columns, which are then backfilled with nulls), these operations still require the types and structures of provided columns to match the existing dataset exactly [6]. - Legacy/Storage Version Issues: Mismatches can sometimes arise from changes in internal storage versions or when internal field IDs do not align correctly during complex schema evolutions [7][1]. To troubleshoot or resolve these errors: - Inspect the schema of the incoming batch (batch.schema) and compare it explicitly against the dataset schema (dataset.schema()) to identify the exact field causing the mismatch [2][3]. - Ensure that the column order and types match, even if field names are correct [3]. - If you are adding new columns or changing existing ones, use dedicated evolution methods like add_columns or merge instead of attempting a direct append of a mismatched schema [8][9][10]. - Check for library versions; some issues related to schema metadata have been fixed in recent releases (e.g., those involving pa.json_() handling) [4][5]. If you are performing schema changes (like adding columns), it is recommended to do so when no other concurrent write operations are taking place, as schema evolutions may conflict with active appends [9][10].

Citations:

• 1: https://github.com/lancedb/lance/blob/119f87b3/rust/lance/src/dataset/write.rs
• 2: LanceDBError: Append with different schema lancedb/lancedb#449
• 3: bug(node, lancedb): unable to 'add' to a table created with a Float64 vector lancedb/lancedb#1293
• 4: bug(rust): table.add() fails with schema mismatch when appending PyArrow JSON extension type data lancedb/lancedb#3144
• 5: Bug(rust): dataset.update() fails with schema mismatch on datasets containing pa.json_() columns lance-format/lance#6329
• 6: feat: allow inserting subschemas lance-format/lance#3041
• 7: feat: support complex schemas in append lance-format/lance#2209
• 8: https://docs.lancedb.com/tables/schema
• 9: http://lance.org/guide/data_evolution/
• 10: https://lance.org/guide/data_evolution/

🌐 Web query:

"pub struct Schema" "fields: Vec<Field>" "lance" "9.0.0"

💡 Result:

In the Lance library (specifically within the lance-core crate), the Schema struct is defined as follows: pub struct Schema { pub fields: Vec, pub metadata: HashMap<String, String>, } This structure represents the top-level schema of a Lance dataset [1]. The fields vector contains the top-level Field definitions for the table [1]. The Field struct, which defines individual columns or nested fields, typically includes the following members [2][3][4]: - name: String - id: i32 - parent_id: i32 - logical_type: LogicalType - metadata: HashMap<String, String> - encoding: Option - nullable: bool - children: Vec (or Vec depending on version) - dictionary: Option - unenforced_primary_key_position: Option - (In some versions) unenforced_clustering_key_position: Option Lance uses this schema information to manage columnar storage, data versioning, and ACID-compliant schema evolution [5][6]. While Schema is used internally in the Rust implementation, users typically interact with schemas through high-level APIs in Python, TypeScript/JavaScript, or Rust that leverage Apache Arrow for data definition [7].

Citations:

• 1: https://github.com/lancedb/lance/blob/119f87b3/rust/lance-core/src/datatypes/schema.rs
• 2: https://github.com/lancedb/lance/blob/119f87b3/rust/lance-core/src/datatypes/field.rs
• 3: https://docs.rs/lance/latest/lance/datatypes/struct.Field.html
• 4: https://docs.rs/lance-core/latest/lance_core/datatypes/struct.Field.html
• 5: https://docs.lancedb.com/tables/schema
• 6: https://docs.lancedb.com/lance
• 7: https://docs.lancedb.com/tables

🌐 Web query:

docs.rs lance 9.0.0 Dataset schema .fields .field

💡 Result:

In the Lance library (referenced in version 9.0.0 via the lance crate), the Dataset struct and its associated Schema structure provide mechanisms for accessing field information [1][2]. The Dataset struct has a method .schema() which returns a reference to the Schema object associated with the dataset [1]. The Schema struct defines the dataset's fields in the following way [2]: pub struct Schema { pub fields: Vec, pub metadata: HashMap<String, String>, } Key methods available on Schema to interact with fields include [2]: - .field(name: &str) -> Option<&Field>: Retrieves a field by its path (dot-separated for nested fields). Field names containing dots must be quoted (e.g., parent."child.with.dot"). - .field_id(column: &str) -> Option<i32>: Retrieves the ID of a field by its path. Additionally, the Field struct (defined in lance::datatypes) represents an individual column and provides its own methods for navigating and interacting with the schema, such as .sub_field(path_components: &[&str]) and .field_by_id(id: impl Into<i32>) [3][4].

Citations:

• 1: https://docs.rs/lance/latest/lance/dataset/struct.Dataset.html
• 2: https://github.com/lancedb/lance/blob/119f87b3/rust/lance-core/src/datatypes/schema.rs
• 3: https://docs.rs/lance/latest/lance/datatypes/struct.Field.html
• 4: https://docs.rs/lance-core/latest/lance_core/datatypes/struct.Field.html